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Rsync: 




Tutorial and Examples to Copy, Mirror, Synchronize, Backup 
and Restore files 



Rsync is a mechanism to mirror data on a single host or between multiple 
hosts. 

Rsync can duplicate directories and files and provide backup/restore 
operations or mobile device synchronization. This tutorial covers Rsync client 
and server configurations for both Linux and MS/Windows (using Cygwin). 



Rsync description, topologies and configuration: 

Rsync can duplicate data between two directories whether the data is collocated on the same computer or over the 
network between two computers. The directories can be on any accessible random access storage devices (hard 
drives, thumb drives, etc). The data source must be readable and the destination must be writable. 

The benefit of Rsync is the ability to update the mirrored backup with computed changes from the data source with 
a minimal load to system resources. It is very data bandwidth and system intensive to perform an entire copy of a 
directory structure each time a backup is to be performed. While the initial backup is no faster than a copy, Rsync 
has the ability to load the system in a minimal fashion by transferring only changes when subsequent rsync 
updates are performed. 

Description of the rsync algorithm 

Rsync single host: mirror between storage devices 




Rsync client-server: mirror between two computers 




Rsync Single Host Operation: 

Rsync command format: rsync [options] source-path destination-path 
Synchronize directory paths each of which are accessible to a single system. 
Examples: 



• Back-up (push) your photo directories to a second drive: 

rsync -av -/Pictures /mnt/drive2 

This creates a backup of your photos in /mnt/drive2/Pictures/ 
Back-up to a USB thumb drive is similar: rsync -av -/Pictures /media/KlNGSTON 
When you add new photos, just re-execute this rsync command to backup the latest changes. 
Note: The drive name will be dependent on the manufacturer. 

[Potential Pitfall]: Do not include the source directory name in the destination. 

rsync -av -/Pictures /mnt/drive2/Pictures 

This will result in /mnt/drive2/Pictures/Pictures/ 

Note that rsync destinations acts just like the cp and rep commands. 

Also note that rsync -av -/Pictures/ /mnt/drive2/ has a different behavior from rsync -av 
-/Pictures /mnt/drive2 

• Back-up (push) two directory source to a single directory on the second drive: 

rsync -av -/Pictures -/Images /mnt/drive2 

This creates a backup of your photos from the two directories in /mnt/drive2/Pictures/ 

• Sync directories and if any were deleted in -/Pictures, also delete it in /mnt/drive2/Pictures/: 
rsync -a --progress --delete -/Pictures /mnt/drive2 

This creates a backup in /mnt/drive2/Pictures/ 

• Sync one specific file in a directory: 

rsync -a ~/Pictures/Group-photo-2001-A. jpg /mnt/drive2/Pictures 

• Sync a group of specific files in a directory: 

rsync -a -/Pictures/2001-* . jpg /mnt/drive2/Pictures 
This creates a backup in /mnt/drive2/Pictures/ 

Note that when transferring files only, the directory name has to be provided in the destination path. 

• Sync files and directories listed in a file: 

rsync -ar --f iles-f rom=Filelist . txt -/Data /mnt/drive2 
This creates a backup in /mnt/drive2/Data/ 

Directory paths are included if specified with a closing slash such as pathx/pathy/pathz/. Path names must 
be terminated with a 7" or 7." 

• Back-up (push) your source code and compress to save space. Ignore object files: 

rsync -avz --delete --exclude^ ' * . o ' --exclude= ' * . so ' ~/src /mnt/drive2 

This creates a backup in /mnt/drive2/src/ but does not transfer files with the ".o" and ".so" extensions. 

• Back-up (push) your source code and ignore object and shared object code files: 

rsync -av --delete --filter='+ *. [ch] ' — filter='- *.o' ~/src /mnt/drive2 

This transfers files with the extension ".c" and ".h" but does not transfer object files with the ".o" extensions. 

sameasrsync.exe -av --exclude= ' * . o ' --filter='+ *.c *.h' ~/src /mnt/drive2 

• Back-up (push) your source code and ignore CM directories, object and shared object code files: 

rsync -artv --progress --delete --filter='+ *. [ch] ' ~-filter='- *.o' — 
exclude=" . svn" ~/src /mnt/drive2 

Note that — exclude overides the include filter — filter='+ *. [ch] ' so that ".c" and ".h" files under 
. svn/ are not copied. 

Referencing directories - errors and subtleties: 

• rsync -ar dirl dir2 

This will copy dirl into dir2 to give dir2/dirl thus dirl/f ile . txt gets copied to 

dir2/ dirl/ file. txt 



• rsync -ar dirl/ dir2/ 

This will copy the contents Of dirl into dir2 to give dir2/contents-of-dirl, eg dirl/f ile . txt 

gets copied to dir2/f ile . txt 

The following all achieve the same results: 

o rsync -ar dirl/ dir2/ 
o rsync -ar dirl/* dir2 
o rsync -ar dirl/* dir2/ 

Rsync Options: (partial list) 



Command 
line argument 


Description 


-a 

(-archive) 


Archive. 

Includes options: 

• -r: recursion 

• -1: preserve symbolic links as symbolic links. Opposite of -L 

• -p: preserve permissions (Linux/unix only) 

• -t: preserve file modification time 

• -g: preserve group ownership 

• -o: preserve user ownership 

• -D: preserve special files and devices (Linux/unix only) 


-d 

(»dirs) 


Copy directory tree structure without copying the files within the directories 


—existing 


Update only existing files from source directory which are already present at the 
destination. No new files will be transferred. 


-L 

(-copy-links) 
-stats 


Transform a symbolic link to a copied file upon transfer 


Print verbose set of statistics on the transfer 

Add -h ( — human-readable) to print stats in an understandable fashion 


-P 

(-perms) 


Preserve permissions (not relevant for MS/Windows client) 


-r 

(-recursive) 


Recursive through directories and sub-directories 


-t 

(-times) 


Preserve file modification times 


-V 

(-verbose) 


Verbose 



-z 

(-compress) 


Compress files during transfer to reduce network bandwidth. Files not stored in an 
altered or compressed state. 

Note that compression will have little or no effect on JPG, PNG and files already using 

compression. 

Use arguments — skip- 

compress=gz/bz2/ jpg/ jpeg/ ogg/mp [34 ] /mov/ avi/rpm/deb/ to avoid 
compressing files already compressed 


-delete 


Delete extraneous files from destination directories. Delete files on archive server if they 
were also deleted on client. 

Use the argument -m ( — prune-empty-dirs) to delete empty directories (no longer 
useful after its contents are deleted) 


-include 
-exclude 
— Tiiter 


Specify a pattern for specific inclusion or exclusion or use the more universal filter for 
inclusion (+)/exclusion (-). 

Do not transfer files ending with ".o": — exclude= ' * . o ' 
Transfer all files ending with ".c" or ".h": — filter='+ *. [ch] ' 


-i 

(-itemize- 
changes) 


Print information about the transfer. List everything (all file copies and file changes) 
rsync is going to perform 


-list-only 
-dry-run 


Don't copy anything, just list what rsync would copy if this option was not given. This 
helps when debugging the correct exclusion/inclusion filters. 


-progress 


Shows percent complete, Kb transferred and Kb/s transfer rate. Includes verbose 
output. 



For all options see the rsync man page 



Note that rsync will be able to handle files with blanks in the file name or directory name as well as with dashes ("- 
") or underscores ("_")■ 



Rsync Client-Server Configuration and Operation: 

Rsync can be configured in multiple client-server modes. 

1 . connect client to a sever running rsync in daemon mode 

2. connect client to a sever using a ssh shell 

These configurations are specified with the use of the colon ":" 

• Double colon refers to a connection to a host running the rsync daemon in the format 
hostname::module/path where the module name is identified by the configuration in /etc/rsyncd.conf. The 
double colon is equivalent to using the URL prefix rsync : // 

• Single colon refers to the use of a remote shell 

• No colon then the directory is considered to be local to the system. 



1) Rsync daemon server: 

The Rsync server is often referred to as rsyncd or the rsync daemon. This is in fact the same rsync executable run 
with the command line argument " — daemon". This can be run stand-alone or using xinetd as is typically 
configured on most Linux distributions. 



Configure xinetd to manage rsync: 
File: /etc/xinetd.d/rsync 

Default: "disable = yes". Change to "disable = no" 

01 service rsync 

02 { 

03 disable = 
no 

04 flags 
IPv6 

05 socket_type = 
stream 

06 wait = 
no 

07 user = 
root 

08 server = 
/usr/bin/ rsync 

09 server_args = -- 
daemon 

10 log_on_f ailure += 
USERID 

11 } 



Start/Re-Start xinetd: /etc/rc . d/init . d/xinetd restart 
For more information on xinetd see the YoLinux xinetd tutorial. 

Typical Linux distributions do not pre-configure rsync for server use. Both Ubuntu and Red Hat based distributions 
require that one generates the configuration file "/etc/rsyncd. conf" 

File: /etc/rsyncd. conf 



01 log file = 

/var/log/rsyncd . log 



02 hosts allow = 192.17.39.244, 
192.17.39.60 



03 hosts deny = 
* 



04 list = 
true 



05 uid = 
root 

06 gid = 
root 



07 read only = 
false 

08 



09 [Projl] 

10 path = 

/ tmp/Pro j 1 



11 comment = Project 1 rsync 
directory 



13 [ProjX] 

14 path = 
/var/Pro jX 



15 comment = Project X rsync 
directory 



rsyncd.conf man page 



Client command to rsync to the server: 

Push: rsync -avr /home/userl /Pro j 1/Data server-host-name :: Pro j 1 
(eg. update server backup from mobile laptop) 

This will initially copy over directory Data and all of its contents to /tmp/Proj l/Data on the remote server. 

Pull: rsync -avr server-host-name :: Pro j 1 /home/userl/Proj 1/Data 
(eg. update mobile laptop from server backup) 



2) Rsync to server using ssh shell: 

Using this method does not use the configuration "modules" in /etc/rsyncd. conf but instead uses the paths as 
if logged in using ssh. 

First configure ssh for "password-less" login: 

Note that current Linux distributions use ssh version 2 and rsa. 



[userl@myclient ~]$ ssh-keygen -t rsa 
Generating public/private rsa key pair. 

Enter file in which to save the key (/home/userl/ . ssh/id_rsa) : 
Enter passphrase (empty for no passphrase) : 
Enter same passphrase again: 

Your identification has been saved in /home/userl/ . ssh/id_rsa . 
Your public key has been saved in /home/userl/ . ssh/id_rsa . pub . 
The key fingerprint is: 
aa:lc:76:33:8a:9c:10:51: 



Note that "Enter" was pressed when asked for a "passphrase" to take the default. 
Two files are generated: 

• Local client (private key): ~/.ssh/id_rsa 

• Contents (one line) of file (public key): ~/.ssh/id_rsa.pub to be copied into file on server: 
~/.ssh/authorized_keys 

Note file protection on file: 

[userl@myclient ~] $ Is -1 ~/ . ssh/id_rsa 

-rw . 1 userl userl 1675 Sep 7 14:55 

/home/userl/ . ssh/id rsa 



Copy public key to server so you can login. 

Use the following command: Now try logging into the machine, with "ssh 'userl @remote-server"', and check 
in: .ssh/authorized_keys to make sure we haven't added extra keys that you weren't expecting. 

Test "password-less" ssh connection: ssh remote-server 
This command should log you in without asking for a password. 

Now try rsync (push) using ssh: 



rsync -avr --rsh=/usr/bin/ssh /home/userl /Pro j 1 /Data remote- 
server : /mnt/ supers an /Pro j 1 

Note that if this connection is is to be spawned by a cron job (eg. root user) then the shell user ID must be 
provided: useri@ 

rsync -avr --rsh=/usr/bin/ssh /home/userl /Pro j 1 /Data userl@remote- 

server : / mnt/ supers an /Pro j 1 

SSH options may be put in the file ~/ . ssh/conf ig 
crontab: 

Note that rsync is often used with cron to perform a nightly rsync. 
eg. Rsync to get latest updates to the web server at 2:00am: 
File: /etc/crontab 

1 * 2 * * * rsync -avr server-host-name :: Proj 1/html /var/www > 
/var/log/rsync 2>&1 



See the crontab man page 
MS/Windows Rsync Client and Server Operation: 

Multiple solutions exist for MS/Windows rsync clients but they all rely on Cygwin. There is no other port of rsync. 

• Cygwin: Work in Cygwin shell environment 

• DeltaCopy (client and server) 
Uses Rsync algorithm and protocol 

• QtdSync: (client and server) MS/Windows version includes Cygwin rsync and ssh (and more) 
[download] 

Cygwin rsync client for MS/Windows: 

Cygwin must be installed for the purpose of this tutorial. 
Be sure to install the Cygwin component "Net" + "rsync". 

Rsync Cygwin components: 



C : \cygwin\bin\rsync . exe 
\ssh . exe 
\sshpass . exe 
\cygcrypto- 

0.9. 8 .dll 

\cyggcc_s-l . dll 
\cygiconv-2 . dll 
\cgypopt-0 . dll 
\cygssp-0 . dll 
\cygwinl . dll 
\cygz . dll 

These components can be extracted for a minimal installation of be left in place as part of the complete 
Cygwin package. 

DOS bat script to perform an rsync (push): 



1 http : //twitter . com/set">@set 
PATH=C : \cygwin\bin 

2 http : //twitter . com/set">@set 
LIB=C : \cygwin\bin 

3 rsync.exe -avr /cygdrive/c/MISC/ JUNK remote- 
server : : Pro j 1 



[Potential Pitfall]: The following command: rsync . exe -avr c:MISC\JUNK remote-server: : Pro jl will 
produce an error because rsync will interpret the "C:" as the remote host and not a drive letter. Drive letters are not 
POSIX compliant. 

[Potential Pitfall]: The following commands: 

cd c : \ 

rsync.exe -avr MISC\JUNK remote- 
server : : Pro j 1 

will generate a single directory of the name 'MISC\JUNK' (not two directories, one within the other but just one with 
that name). The 7" will not be a directory delimiter but a character in the directory name (bad). 

For more on Cygwin POSIX paths: http://cygwin.eom/cygwin-ug-net/using.html#using-pathnames 



MS/Windows rsync client GUI: 

I chose QtdSync/cygwin as it provided a GUI and an isolated Cygwin rsync executable and its dependencies. 
QtdSyncGUI: 



QtdSync v0.6. 1 5 beta - New backup set.qtd* 



QtdSync v 0.6. 15 beta 

© 20 1 1 by Thomas Doting 



A \ 
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E3 


0 


• 


• 


File, 


Schedule 


Notification 


Run Backup 


Settings 


About QtdSync 



Name 
Host 



New backup set 



v 



Synchronize Only v Q Backup into subfblder 



rsync://megacorp.com Q Path Projl 



v & User 



Last Backup Unknown 

Tasks 



Elements 

a Q junk 

S Q C:/MISC/JUNK (bound to this computer) 



0 



Report a bug! 



0 Back to Startup Q Save & Quit Q Cancel 



To list GUI command line arguments: c : \Program Files\QtdSync\QtdSync . exe --help 
Includes rsync Cygwin components: 



C : \ Program Files\QtdSync\bin\rsync.exe 

\ssh . exe 
\sshpass . exe 
\cygcrypto- 

0.9. 8.dll 

\cyggcc_s-l . dll 
\cygiconv-2 . dll 
\cgypopt-0 . dll 
\cygssp-0 . dll 
\cygwinl . dll 
\cygz . dll 
\etc\f stab 



Where the file C : \Program Files\QtdSync\etc\f stab has the following contents: 



none /cygdrive cygdrive binary, posix=0 , user , noacl 0 
0 



Note: QtdSync does not require the separate installation of cygwin. 



DOS bat script to perform an rsync (push): 



1 http : //twitter . com/set">@set PATH=C : \Program 
Files \QtdSync\bin 

2 http : //twitter . com/set ">@set LIB=C : \Program 
Files \QtdSync\bin 



3 rsync.exe -avr /cygdrive/c/MISC/ JUNK remote- 
server : : Pro j 1 



ACygwin MS/Windows rsyncd server configuration: 

Rsync daemon configuration requires installing rsync as a MS/Windows service. Use the Cygwin command 
cygrunsrv command (installed under the Cygwin "Admin" category). 

cygrunsrv command options: 



Option 


Description 


-1 

-install 


Install as a service 


-R 

-remove 


Remove as a service 


-S 

-start 


Start the service 


-E 

-stop 


Stop the service 


-a 

-args 


Arguments to pass the daemon service 


-h 


Command help instructions 



Install rsync as a MS/Windows service: regrsyncd.bat 



1 http : //twitter . com/set">@set 
PATH=C : \cygwin\bin 



2 http : //twitter . com/set">@set 
LIB=C : \cygwin\bin 



3 cygrunsrv -I rsyncd -e CYGWIN=nontsec --path 
/ cygdrive/ c/ cygwin/bin/ rsync . exe \ 

4 -a "--daemon --conf ig=/etc/rsyncd. conf --no- 
detach" 



You may be required to include the system user login and password: 

cygrunsrv -I rsyncd -e CYGWIN=nontsec --path /cygdrive/c/cygwin/bin/rsync . exe 
\ 

-a "--daemon --conf ig=/etc/rsyncd . conf --no-detach" -u 
Administrator -w super-secret-password 



Note that Cygwin applications refer to configuration file referenced within the Cygwin environment. As 
viewed by the Cygwin rsync application, the file /etc/rsyncd. conf is actually 

c : \cygwin\etc\rsyncd. conf as viewed by the MS/Windows OS. 

Where the option -e CYGWiN=nontsec turns off Microsoft Windows NT security permissions. On 
MS/Windows 7 use the following c : \ cygwin\etc\ f stab settings: 



1 none /cygdrive cygdrive binary, posix=0 , user , noacl 0 

0 



fstab man page 

This is to avoid the following error: 

rsync: failed to modify permissions on xxxxf ilenamexxxx : Permission denied 
File: C : \cygwin\etc\rsyncd . conf 



01 log file = 

/var/log/rsyncd 



02 hosts allow = 192.17.39.244, 
192.17.39.60 



03 hosts deny = 
* 



04 list = 
true 

05 use chroot = 
false 



06 strict modes = 
false 



07 read only = 
false 

08 ignore nonreadable = 
yes 



09 dont compress = *.gz * . tgz *.zip * . rpm * . deb *.iso *.bz2 *.jpg *.mpg 
* . mpeg 

10 

11 [Projl] 

12 path = 

/ cygdrive/c/Pro j 1 



13 comment = Project 1 rsync 
directory 



rsyncd.conf man page 

Where the Cygwin log file /var/log/rsyncd is C : \cygwin\var\log\rsyncd as viewed by the 
MS/Windows OS. 

The Cygwin rsync directory path /cygdrive/c/Proj 1 isC:\Projl\ as viewed by the MS/Windows OS. 



Check to see if the rsyncd service is running: Control Panel — > Administrative Tools — > 
Services 



% Services ^ 



File Action View Help 



oh & m © 



Services (Local) 



Services (Local) 



rsyncd 

Stop the service 
Restart the service 



Extended 



Name 


Description 


Status 


Startup 


^Remote Registry 


Enables re... 


Started 


Automatic 


^Removable Storage 




Started 


Automatic 


Routing and Remot. . . 


Offers rout. . . 




Disabled 


L ^rsyncd 




Started 


Automatic 


Secondary Logon 


Enables st. . . 


Started 


Automatic 


^Security Accounts ... 


Stores sec. . . 


Started 


Automatic 


^Security Center 


Monitors s... 




Disabled 


% Server 


Supports ft... 


Started 


Automatic 


% Shell Hardware Det... 




Started 


Automatic 


% Smart Card 


Manages a... 




Manual 



< 



Find "rsync". If not running, select the service and select "start" or issue the command "net start 
rsync". 

[Potential Pitfall]: You must check to see if the service has started. If not select " Start the service". 

The Microsoft Windows firewall may not allow a system to attach to rsyncd on port 873. My experience is that the 
service is detected by the OS and added to the firewall privileges. I did not have to configure the firewall, 
registration was automatic. If the rsyncd service did not get registered with the MS/Windows firewall, one may have 
to specifically grant the permission to to the rsyncd service. 

Select "Control panel" + Windows Firewall": 



Windows Firewall 



□ 



General Exceptions Advanced 



For your security, some settings are controlled by Group Policy 



Windows Firewall helps protect your computer by preventing unauthorized users 
Prom gaining access to your computer through the Internet or a network. 



On (recommended) 

This setting blocks all outside sources from connecting to this 
computer, with the exception of those selected on the Exceptions tab. 



f 



n Don 1 alio w except i ons 

Select this when you connect to public networks in less secure 
locations, such as airports. You will not be notified when Windows 
Firewall blocks programs . Selections on the Exceptions tab will be 
ignored. 

Off (not recommended) 

Avoid using this setting. Turning off Windows Firewall may make this 
computer more vulnerable to viruses and intruders. 



Windows Firewall is using your domain settings 
What else should I know about Windows Firewall 7 



OK 



Cancel 



Windows Firewall 



General &<ceptions [Advanced 



Windows Firewall is blocking incoming network connections, except for the 
programs and services selected below. Adding exceptions allows some programs 
to work better but might increase your security risk. 



Programs and Services: 



Name Group Policy 




0 RedHatLinux 






0 Remote Assistance 


No 




0 Remote Desktop 


Yes 




0 Retrospect 


Yes 




0 Retrospect 


Yes 




0 Rhapsody 


No 




BPIrsync 


No 




0rtiexec 


No 




0 Sametime 


Yes 




0 Sametime 


Yes 













Add Program... 


Add Port... 


Ed. 


Delete 



[^1 Display a notification when Windows Firewall blocks a program 
What are the risks of allowing exceptions? 



OK 



Cancel 



Edit a Program 



You can allow communication with this program from any computer, 
including those on the Internet or just from computers on your network. 



Name: ""Irsvnc 



Path: C:\Program Files\QtdSync\bin\rsync.exe 



Change scope. 



OK 



Cancel 




To specify the set of computers for which this port or program is unblocked, click an 
option below. 



To specify a custom list, type a list of IP addresses, subnets, or both, separated by 
commas. 



0jAf!Y ^computer including those m 



O My network (subnet) only 



( J Custom list: 



Example: 192.168.114.201.192.168.114.201/255.255.255.0 



OK 



Cancel 



To remove rsyncd from the available list of services, execute the following command from a "cmd" terminal: 

c:\cygwin\bin\cygrunsrv -R rsyncd 



Debugging Rsync: 

Using telnet to test an rsync connection: 

One can use telnet to test connectivity with an Rsync server: 

[bash]$ telnet 192.168.1.12 873 
Trying 1 92 . 1 68 . 1 . 12 . . . 
Connected to 192.168.1.12. 
Escape character is IA ] ' . 
0RSYNCD: 30.0 
A C 

Connection closed by foreign 

host . This is the proper response when using telnet to connect to 



This is representative of a telnet connection to a misconfigured rsync server or one which has not yet been started 
(nothing for the client to connect to): 

[bash]$ telnet 192.168.1.13 873 
Trying 1 92 . 1 68 . 1 . 13 . . . 

telnet: connect to address 192.168.1.13: Connection timed 
out 



rsync. 



does not get past this 



and times out. 



Rsync client responses: 

An rsync server which has been shut down or was never started will cause the rsync client to behave as follows: 

[bash] $ rsync -avr 1 92 . 1 68 . 1 . 12 : : Pro j 1 /home/userl/Test 
rsync: failed to connect to 192.1.1.12: Connection timed out (110) 
rsync error: error in socket 10 (code 10) at clientserver . c ( 124 ) 
[ receiver=3 .0.6] 



Rsync logs: 

Check the rsyncd log file: /var/log/rsyncd. log 

2011/09/08 21:13:10 [1032] rsyncd version 2.6.6 starting, listening on port 
873 

2011/09/08 21:17:10 [3156] rsync error: received SIGUSR1 or SIGINT (code 20) 
compared to normal logging of a client connection and file transfer: 

2011/09/08 21:19:10 [2796] rsync on Projl/ from 192.168.1.10 

2011/09/08 21:19:10 [2796] sent 940 bytes received 334 bytes total size 

97 



Links: 

• Samba Rsync home page 

• Lsyncd - Unix/Linux only. Monitor drive for changes (inotify) and perform an rsync any time a change is 
made. 

Public rsync servers: 

• rsync.net 

• s3rsync.com 
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